Synthetic vision and the closely related field of augmented reality encompass techniques and devices that give people an enhanced understanding of their surroundings through visual displays. Synthetic vision is an improvement over normal vision because it uses digital graphics to let people see things that would otherwise be invisible: a runway shrouded in fog, a buried pipeline, or a proposed building yet to be built.
A synthetic vision display may show: (I) actual real-time images such as live video or radar, (II) digitally created real-time images such as computer graphics, or combinations of (I) and (II). The combined presentation of actual and digitally created images on a display recreates the visible and invisible world as one travels through the air, on the earth's surface or under the sea. The combination of real and recreated views is sometimes called augmented reality. Here “synthetic vision” and “augmented reality” are used interchangeably.
Although the possibility of synthetic vision has been discussed for years, actual usable systems have only recently emerged as compact computing, attitude sensing and position sensing technology have matured. It has recently become possible, for example, to install a basic synthetic vision system in an aircraft although such systems are still in the development phase. A synthetic vision system in an aircraft cockpit may show pilots an accurate, three-dimensional view of desired flight paths, traffic, runways, terrain and obstacles near the aircraft. Research has shown that synthetic vision is easy for pilots to learn and use.
Azuma proposed an “optical see-through augmented reality modified-scale display” in US 2004/0051680 incorporated herein by reference. Azuma uses an optical approach where a user directly sees the real world through optics with graphics optically merged in. According to Azuma, “a person looking through a pair of binoculars might see various sights but not know what they are. A soldier could look through a pair of augmented binoculars and see electronic battlefield information directly superimposed upon his view of the real world (labels indicating hidden locations of enemy forces, land mines, locations of friendly forces, and the objective and the path to follow). A spectator in a stadium could see the names of the players on the floor and any relevant information attached to those players . . . ” Lynde, U.S. Pat. No. 6,181,302 incorporated herein by reference, describes a similar augmented reality binocular system for marine applications.
The multimedia group at VTT Information Technology in Espoo, Finland has built an augmented reality system using a personal digital assistant as the display. See “Implementation of an Augmented Reality System on a PDA”, W. Pasman and C. Woodward, Proc. ISMAR 2003, Tokyo, Japan, 4-7 Nov. 2003 incorporated herein by reference. The software runs in a client/server model and calculations are distributed between a server laptop and the PDA, using a WLAN link. Among other tasks, the software uses ARToolkit to track markers placed in the scene.
Surveylab Group Limited, Wellington, New Zealand offers the “IKE” line of rapid data capture devices. IKE combines GPS positioning, a digital camera and a laser range finder in a handheld device that records position-tagged digital photographs.
Ellenby proposed an “electro-optic vision system which exploits position and attitude” in U.S. Pat. No. 5,815,411 incorporated herein by reference. Ellenby mentions a planoptic approach to retrieving display data in which “the data store can have pre-programmed information regarding particular scenes that are of interest to a specific user. The position determining means and the attitude determining means control the pointer of the data store.”
Conventional approaches to synthetic vision include undesirable limitations. Some of these are due to lack of computing power. Performing three dimensional graphics is computationally expensive. Each point in a graphics database—including vertices of all lines and polygons—must undergo a perspective transformation and projection using floating-point 4×4-matrix arithmetic. Additional integer math is required to actually draw polygons or lines on a screen. High bandwidth to video memory is required to achieve sufficient frame refresh rates. It has been estimated that a person using a calculator to perform all the computations in a complex scene could complete only two frames in his lifetime!
A new generation of 3D rendering chips, developed in response to demand from PC multimedia and gaming markets, are making sophisticated graphics possible at favorable price/performance levels. The power of a $10,000 graphics machine ten years ago can now be found on a single graphics chip or even incorporated into a microprocessor itself. In 1994 state-of-the-art mobile computing was represented by an 80486-based laptop with DOS as the operating system. Graphics chipsets had no 3D acceleration capabilities and display frame rates were limited to 12˜15 Hz. By 1997 better operating systems and chipsets such as the GLiNT 500TX could achieve graphics frame rates as high as 20 Hz. Today, inexpensive mobile computers routinely incorporate microprocessors running as fast as 1 GHz, high resolution active matrix liquid crystal display screens and high resolution graphics with 20 Hz or greater update rate capability.
Another key problem in conventional synthetic vision is registration. Registration refers generally to displaying actual and digitally created objects in proper position relative to one another. Registration errors have many sources including inaccuracy in knowing one's position and, in some applications, inaccurate modeling of the curvature of the earth's surface. Some conventional synthetic vision systems use special fiduciary marks placed in a real scene to help overcome registration errors. A mark, such as a large letter “A” in a known location, can be recognized by image processing software. Since the mark's location is already known, digitally created objects can be aligned to it. The registration problem is especially acute for high magnification systems and for optical see-through devices.
Some conventional synthetic vision systems use a planoptic function approach to computer graphics. In this approach scene data for each possible position and attitude of a device is prerecorded in memory. The current position and attitude of the device are then used to select scenes from memory. Since it is desirable for the device to be able to display scenes corresponding to many different positions and attitudes, a large memory capacity is needed.
What is needed is a synthetic vision system that is small and light enough to be easily hand held for personal use or compactly mounted in a vehicle. Desirable attributes of such a system would include precise positioning, reduced registration errors and graphics rendering at 20 Hz or greater frame rates. Ideally the system should include compact attitude and heading sensing devices.
In addition it would be desirable to extend synthetic vision to include display of actual and/or digitally created scenes from arbitrary view points. For example, it would be useful for a construction engineer to be able to see a proposed building or a buried water pipe from a variety of perspectives: his current location, from overhead, from 45 degrees off to one side, etc.
It would also be desirable for the system to use a modern graphics rendering scheme in which data is culled from a database of objects according to their spatial locations and the extent of a computed view frustum. Culling allows the computer to quickly identify large blocks of data that can be ignored for a particular scene. In contrast to a planoptic approach, objects need only appear in a database once.